Network security is becoming increasingly important as the information age continues to unfold. Network threats/attacks may take a variety of forms, including unauthorized requests or data transfers, viruses, malware, large volumes of traffic designed to overwhelm resources, and the like. A variety of automated cyber analysis systems have been developed to protect networks against such network threats. In practice, cyber analysis systems are often operated in a highly inefficient manner.
Conventional network protection solutions, such as network firewalls and network intrusion detection systems, are often too inefficient and slow to actively and proactively protect networks from modern Internet-borne cyber threats and attacks; thus, those solutions are unable to effectively protect enterprise networks. These systems detect network threats by analyzing network communications using signature-based methods, anomaly-based methods, behavioral-based methods, intelligence-based methods, malware analysis methods, and the like. Often, these cyber analysis systems are used to reactively defend networks, e.g., detecting and mitigating threats/attacks after they have occurred.
In TCP/IP networks, a communication is a (usually bi-directional) flow of packets between two endpoints, and may be characterized by the L3/L4 “5-tuple” of source and destination IP addresses, source and destination ports, and L4 protocol type (e.g., TCP, UDP, etc.). Conventional solutions may log all packet communications crossing the enterprise network perimeter, which may be often located at the boundary between the protected network and the Internet. Packets may also be captured, copied, and/or stored, for use in subsequent cyber analysis. Stored logs may be searched for communications that are potential threats/attacks. The stored packets may be input into automated cyber analysis systems that search for signatures and behaviors that indicate potential threats. Automated cyber analysis systems are usually not deployed as inline systems because they may decrease network performance to unacceptable levels as traffic load increases.
Any potential threats may be reported to human cyber analysts, who either (a) determine that communications may be a threat, and identify any remedial actions that may mitigate the threat; or (b) determine that the communications may not be a threat; or (c) make no determination because, for example, there may be insufficient information to make a determination, or, for example, they do not investigate a threat due to the overwhelming volume of potential threats in their work queues.
Because confirmed threats/attacks typically represent less than 1% of the volume of enterprise communications with the Internet, a conventional solution approach can be highly inefficient, slow, and inaccurate. The large majority of available time and resources may be wasted searching through and analyzing legitimate (non-threat, benign) communications. Furthermore, many actual threats are not discovered because the diversity and complexity of threats make it difficult to generate search criteria and analysis rules and algorithms that detect all of them. Search criteria and analysis rules may identify relatively large volumes of legitimate communications as potential threats, which may further increase inefficiencies and inline processing lag (e.g., the time interval spanning the time instant when a threat communications occurred and the time when the threat communications was detected, confirmed, and remediated). Also, increases in the volume and complexity of search criteria and analysis rules may cause significant increases in latency. As a result, despite potentially large capital and operational expenditures on conventional network protection solutions, many actual threats are never discovered, or discovered long after any asset damage, loss, and theft have occurred.
Accordingly, there is a need for efficient and performant operation of cyber analysis systems that will significantly improve the effectiveness of network protection systems. In particular, there is a need for efficient and accurate network protection systems that can actively detect and mitigate threats and associated attacks.